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effort  was  to  involve  scenario  analyses,  sensitivity  analyses  and  external  eval¬ 
uations  by  experienced  DOD  officers  and  corporate  managers.  It  was  also 
intended  to  demonstrate  the  use  of  the  simulation  as  a  management  tool  for  devel¬ 
oping  and  evaluating  alternative  incentive  schemes. 

The  major  practical  use  of  building  a  DPM  type  simulation  is  its  ultimate  appli¬ 
cation  in  answering  "what  if"  type  policy  questions  involving  the  design  param¬ 
eters  of  the  contractual  relationship  between  the  DOD  and  defense  contractors. 

For  example,  the  simulation  results  indicated  that  increasing  the  contractor's 
fee  improves  cost  control  performance.  This  idea  has  recently  been  implemented 
by  the  Navy  with  measurable  success.  Another  example  involves  the  weight  given 
to  past  performance  in  awarding  new  contracts.  The  DPM  simulation  results 
suggest  that  the  higher  the  weight  (including  those  assigned  to  the  quality  of 
the  proposal)  the  better  the  cost  control  performance  and  social  efficiency 
(i.e.,  amount  billed  to  DOD  as  well  as  cost  overruns,  if  any  absorbed  by 
contractor).  Factoring  past  performance  into  the  contract  award  decision  was 
supported  by  the  senior  DOD  officers  interviewed.  The  practical  implications 
involve  the  feasibility  of  factoring  past  performance  measures  into  a  DOD 
contracting  environment  where  often  it  is  difficult  to  maintain  a  second  source 
and  where  such  evaluation  is  considered  to  be  extremely  subjective. 

The  research  effort  demonstrates  the  value  of  such  a  simulation  model  for  anal¬ 
yzing  alternative  acquisition  policies  (e.g.,  the  importance  of  maintaining 
stable  demand,  weighting  past  performance,  etc.)  and  the  design  of  contract 
incentives  (size  of  fee,  schedule  of  progress  payments,  target  and  ceiling  costs 
and  sharing  proportions) .  In  particular  it  could  augment  traditional  analytical 
and  intuitive  approaches  to  these  issues.  Its  future  application  in  acquisition 
policymaking  will  depend  on  implementing  its  use  within  the  relevant  command 
responsible  for  acquisition  policies  and  the  design  of  incentive  contracts. 

A  simulation  model  of  this  type  has  other  uses.  Of  particular  interest  should 
be  its  potential  use  in  the  training  or  education  of  policy  makers  and  or  DOD 
project  managers.  In  business  education  similar  simulation  models  have  been 
designed  as  management  games.  Such  games,  which  can  be  extremely  complex,  are 
used  as  laboratories  for  training  students  to  apply  analytical  tools  and 
integrate  functional  area  knowledge  (marketing,  production,  accounting,  finan¬ 
cial  planning,  etc.)  within  a  competitive  decision  making  environment.  The  DPM 
simulation  has  the  potential  to  be  integrated  into  an  “Acquisition  Game"  and/or 
a  "Project  Management  Game"  for  project  managers.  It  could  also  be  used  in  the 
education  of  policy  makers  (e.g..  Congressmen,  senior  staffs)  when  they  need  to 
develop  insights  into  DOD  acquisition  policies.  The  current  state  of  the  DPM 
simulation  cannot  be  immediately  used  in  acquisition  training.  However,  it 
should  be  at  the  core  of  an  acquisition  game  which  could  be  designed. 

In  sunmary,  the  overall  research  effort  has  resulted  in  a  simulation  model 
involving  the  DOD  and  the  contractor  organization  at  the  corporate  and  project 
management  levels.  The  simulation  has  been  shown  to  be  internally  robust  and  to 
satisfy  certain  face  validity  criteria  as  determined  by  the  external  validation 
interviews.  In  particular  its  potential  application  to  acquisition  related 
policy  making  (e.g.,  design  of  contract  incentives)  has  been  demonstrated.  It 
is  in  the  area  of  contributing  to  policy  making  and  project  management  training 
where  future  developments  need  to  be  directed. 
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EXECUTIVE  SUMMARY 


The  overall  research  project  involved  the  development  of 
a  computer  simulation  model  (DPM)  which  describes  the  organ¬ 
izational  and  contractual  interaction  of  the  defense  con- 
tracting  firm  with  the  DOD  by  focusing  on  the  decision 
process  at  the  project  management  level— — i.e.  ,  decisions 
which  ultimately  determine  the  performance  of  the  contractor 
on  a  given  project.  An  objective  of  this  research  was  to 
develop  a  capability  to  model  the  potential  impact  of  various 
incentive  schemes  on  the  performance  of  defense  contracts . 

It  was  necessary,  therefore,  to  incorporate  in  the  simulation 
model  such  basic  elements  as  DOD  project  goals,  DOD  incentive 
mechanisms,  contractor  goals,  and  contractor  organizational 
response  mechanisms.  Each  of  these  elements,  which  collec¬ 
tively  determine  the  behavioral  pattern  of  the  decision 
process  model  (DPM) ,  are  decoupled  and  parameterized  to 
facilitate  analysis  of  different  incentive  schemes  and/or 
behavioral  assumptions. 

The  objective  of  this  contract  was  to  validate  the  DPM 
simulation  and  its  application  to  developing  and  testing 
alternative  incentive  schemes.  Specifically  this  validation 
effort  was  to  involve  scenario  analyses,  sensitivity  analyses 
and  external  evaluations  by  experienced  DOD  officers  and 
corporate  managers .  It  was  also  intended  to  demonstrate  the 
use  o._  the  simulation  as  a  management  tool  for  developing  and 
evaluating  alternative  incentive  schemes. 


The  major  practical  use  of  building  a  DPM  type  simula¬ 
tion  is  its  ultimate  application  in  answering  "what  if"  type 
policy  questions  involving  the  design  parameters  of  the 
contractual  relationship  between  the  DOD  and  defense  contrac¬ 
tors.  For  example,  the  simulation  results  indicated  that 
increasing  the  contractor's  fee  improves  cost  control  perfor¬ 
mance.  This  idea  has  recently  been  implemented  by  the  Navy 
with  measurable  success.  Another  example  involves  the  weight 
given  to  past  performance  in  awarding  new  contracts.  The  DPM 
simulation  results  suggest  that  the  higher  the  weight  (in¬ 
cluding  those  assigned  to  the  quality  of  the  proposal)  the 
better  the  cost  control  performance  and  social  efficiency 
(i.e. ,  amount  billed  to  DOD  as  well  as  cost  overruns,  if  any 
absorbed  by  contractor) .  Factoring  past  performance  into  the 
contract  award  decision  was  supported  by  the  senior  DOD 
officers  interviewed.  The  practical  implications  involve  the 
feasibility  of  factoring  past  performance  measures  into  a  DOD 
contracting  environment  where  often  it  is  difficult  to 
maintain  a  second  source  and  where  such  evaluation  is  consid¬ 
ered  to  be  extremely  subjective. 

The  research  effort  demonstrates  the  value  of  such  a 
simulation  model  for  analyzing  alternative  acquisition 
policies  (e.g.,  the  importance  of  maintaining  stable  demand, 
weighting  past  performance,  etc.)  and  the  design  of  contract 
incentives  (size  of  fee,  schedule  of  progress  payments, 
target  and  ceiling  costs  and  sharing  proportions) .  In 
particular  it  could  augment  traditional  analytical  and 
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intuitive  approaches  to  these  issues.  Its  future  application 
in  acquisition  policymaking  will  depend  on  implementing  its 
use  within  the  relevant  command  responsible  for  acquisition 
policies  and  the  design  of  incentive  contracts. 

A  simulation  model  of  this  type  has  other  uses.  Of 
particular  interest  should  be  its  potential  use  in  the 
training  or  education  of  policy  makers  and  or  DOD  project 
managers.  In  business  education  similar  simulation  models 
have  been  designed  as  management  games.  Such  games,  which 
can  be  extremely  complex,  are  used  as  laboratories  for 
training  students  to  apply  analytical  tools  and  integrate 
functional  area  knowledge  (marketing,  production,  accounting, 
financial  planning,  etc.)  within  a  competitive  decision 
making  environment.  The  DPM  simulation  has  the  potential  to 
be  integrated  into  an  "Acquisition  Game"  and/or  a  "Project 
Management  Game"  for  project  managers.  It  could  also  be  used 
in  the  education  of  policy  makers  (e.g.,  Congressmen,  senior 
staffs)  when  they  need  to  develop  insights  into  DOD  acquisi¬ 
tion  policies.  The  current  state  of  the  DPM  simulation 
cannot  be  immediately  used  in  acquisition  training.  However, 
it  should  be  at  the  core  of  an  acquisition  game  which  could 
be  designed. 

In  summary,  the  overall  research  effort  has  resulted  in 
a  simulation  model  involving  the  DOD  and  the  contractor 
organization  at  the  corporate  and  project  management  levels. 
The  simulation  has  been  shown  to  be  internally  robust  and  to 
satisfy  certain  face  validity  criteria  as  determined  by  the 
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external  validation  interviews.  In  particular  its  potential 
application  to  acquisition  related  policy  making  (e.g., 
design  of  contract  incentives)  has  been  demonstrated.  It  is 
in  the  area  of  contributing  to  policy  making  and  project 
management  training  where  future  developments  need  to  be 
directed.  • 
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SECTION  1:  INTRODUCTION  AND  BACKGROUND* 

1 . 1  Overview 

A  major  premise  of  incentive  contracts  as  applied  by  the 
DOD  in  the  development  and  procurement  of  new  weapon  systems 
is  that  defense  contracting  firms  are  primarily  motivated  to 
maximize  profits.  Research,  however,  suggests  that  the  goal 
structure  of  contractors  consists  of  survival,  growth,  market 
share,  and  reputation  as  well  as  profit.  Furthermore,  the 
relative  operational  importance  of  these  goals  is  observed  to 
be  significantly  influenced  by  the  position  and  responsibili¬ 
ty  of  the  relevant  decision  maker  within  the  contracting 
organization. 

Survival  is. perceived  by  contractor  management  to  depend 
on  attaining  project  performance  objectives  which  affect 
company  reputation  and  the  ability  to  obtain  future  business. 
Maintaining  quality  technical  and  administrative  personnel, 
even  in  the  face  of  declining  business  activity,  is  also 
thought  to  be  critical  to  securing  future  large-scale  con¬ 
tracts.  Growth  in  market  share  is  pursued  as  a  means  to 
improve  internal  capabilities,  develop  barriers  to  entry  by 
potential  competitors,  and  as  a  means  to  spreading  fixed 


This  section  is  adapted  from  the  "Executive  Summary"  of  the 
Final  Technical  Report  on  Contract  #  F33615-79-C-5151 .  A 
variation  of  this  section  has  been  published  as  Daniel  L. 
Blakeley,  Kalman  J.  Cohen,  Arie  Y.  Lewin,  and  Richard  C. 
Morey,  "Assessing  Defense  Procurement  Policies",  Simulation, 
March  1982,  pp.  75-83.  - 
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costs  over  a  larger  base.  In  short,  contractor  management 
(at  various  levels)  is  observed  to  voluntarily 
sacrifice  short-run  profits  on  a  given  DOD  contract  in  favor 
of : 

a)  improving  opportunities  for  future  follow-on 
projects , 

b)  promoting  technology  spin-offs  to  commercial 
businesses , 

c)  acquiring  or  maintaining  quality  personnel  in 
scarce  disciplines,  and/or 

d)  gaining  competitive  advantage  by  engaging  in 
research  instrumental  to  future  product 
development. 

The  typical  prime  defense  contractor  is  considered  to 
have  a  DOD  business  base  with  an  organizationally  distinct 
commercial  business'  '(see  Exhibit  1.1).  The  defense  business, 
managed  by  a  project  manager  (PM)  ,  consists  of  a  number  of 
projects  awarded  by  the  DOD.  Likewise,  the  commercial 
business  consists  of  a  number  of  product  lines,  some  of  which 
benefit  from  technology  developed  under  DOD  contracts.  In 
addition,  indirect  costs  of  the  commercial  sector  may  poten¬ 
tially  be  transferred  to  DOD  projects  due  to  the  establish¬ 
ment  of  a  larger  overhead  base. 
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Exhibit  1.1 


While  the  commercial  sector  of  the  firm  may  be  based  on 
maximizing  considerations,  the  DOD  business  strategy 
focuses  on  achieving  survival,  growth,  and  prestige  goals 
subject  to  maintaining  a  necessary  level  of  cash  flow.  While 
a  manager  in  the  commercial  market  environment  affects 
revenues  by  control  of  such  variables  as  price,  product 
attributes,  production  levels,  inventory,  etc.,  the  manager 
of  DOD  projects  has  little  operational  control  over  standard 
economic  decision  variables  as  they  are  determined  externally 
via  contractual  negotiations.  Revenue  is  based  on  cost 
recovery;  specifically  on  the  recovery  of  direct  and  indirect 
costs  billed  against  existing  projects.  A  fundamental 
failure  of  past  research  has  been  the  lack  of  a  clear-cut 
understanding  of  the  differences  between  doing  business  in 
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the  traditional  market  environment  and  the  defense  contract¬ 
ing  environment  described  above. 

This  research  project  involved  the  development  of  a 
computer  simulation  model  which  describes  the  organizational 
and  contractual  interaction  of  the  defense  contracting  firm 
with  the  DOD  by  focusing  on  the  decision  process  at  the  PM 
level — i.e.,  decisions  which  ultimately  determine  the  perfor¬ 
mance  of  the  contractor  on  a  given  project.  An  objective  of 
this  research  was  to  develop  a  capability  to  model  the 
potential  impact  of  various  incentive  schemes  on  the  perfor- 
mance  of  defense  contracts.  It  was  necessary,  therefore,  to 
incorporate  in  the  simulation  model  such  basic  elements  as 
DOD  project  goals,  DOD  incentive  mechanisms,  contractor 
goals,  and  contractor  organizational  response  mechanisms. 
Each  of  these  elements,  which  collectively  determine  the 
behavioral  pattern  of  the  decision  process  model  (DPM) ,  are 
decoupled  and  parameterized  to  facilitate  analysis  of  differ¬ 
ent  incentive  schemes  and/or  behavioral  assumptions. 

1.2.  Simulation  Environment 

The  DPM  consists  of  eight  loosely  coupled  sub-models 
which  operate  on  three  organizational  levels:  the  project 
manager  level,  the  corporate  level,  and  the  DOD  level.  Each 
organizational  level  contains  a  separate  set  of  goals, 
expectations,  and  decision  processes  which  are  simulated  by 
the  appropriate  sub-models  (discussed  below) .  Various 
operational  factors  and  project  requirements  are  also  impor¬ 
tant  dimensions  of  the  simulation  environment.  The  synthesis 
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of  these  goals  and  constraints  into  an  operational  plan  by 

the  project  manager  is  the  foundation  of  the  DPM  (see  Exhibit 

1.2)  . 


DOD  Goals 


Exhibit  1.2 

The  DPM  was  designed  to  simulate  the  actions  of  a 
project  manager  in  a  defense  contracting  firm  and  the  inter¬ 
dependent  actions  of  corporate  and  DOD  management  in  a 
dynamic  framework.  As  means  to  simplifying  the  design  of  the 
simulation  (but  consistent  with  accepted  simulation  methods) 
decisions  of  the  project  manager  are  made  on  a  monthly  basis, 
while  corporate  decisions  are  made  quarterly  based  on  the 
project  manager’s  previous  three-month  performance.  The  DOD 
is  also  interested  in  the  project  manager's  actions,  but  only 
as  they  influence  project  performance.  The  DOD  is  able  to 
monitor  specific  project  performance  attributes  on  a  monthly 
(but  delayed)  basis. 
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Contracts  awarded  by  the  DOD  consist  of  heterogeneous 
projects  with  several  distinguishing  attributes.  Each 
project  is  considered  either  a  Major  Project  (e.g.,  a  weapon 
system  awarded  on  a  competitive  basis)  or  a  (significantly 
smaller)  Spinoff  Project  (e.g.,  extension,  enhancement, 
maintenance)  with  a  dollar  budget,  delivery  date,  and  an 
amount  of  standard  quality  of  man-months  (SQMMs)  of  work  to 
be  performed.  In  addition,  Major  Projects  require  that  a 
specific  quality  standard  be  attained  before  the  project  is 
considered  acceptable  for  delivery  to  the  DOD. 

A  primary  goal  of  the  project  manager  (PM)  is  to  control 
the  backlog  of  awarded  projects  within  maximum  and  minimum 
boundaries .  The  mechanism  by  which  man-months  of  backlog  may 
be  influenced  (besides  the  obvious  effects  of  hiring  and 
firing)  involves  the  assignment  of  workers  under  the  PM's 
responsibility  to  indirect  activities.  By  increasing  the 
size  of  the  indirect  work  force,  the  PM  is  able  to  increase 
the  number  of  proposals  submitted  to  the  DOD.  Likewise,  by 
improving  the  quality  of  personnel  writing  proposals,  the  PM 
is  able  to  increase  the  capture  rate  of  the  resultant  propos¬ 
als.  The  quantity  and  quality  of  submitted  proposals,  in 
addition  to  performance  on  previous  contracts,  is  assumed  to 
determine  the  subsequent  awarding  of  new  projects  by  the  DOD. 
Thus,  by  increasing  the  size  or  quality  of  the  indirect 
(proposal  writing)  work  force,  the  PM  is  able  to  increase  the 
number  of  awarded  projects  and,  all  else  equal,  the  man- 
months  of  backlog. 
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A  secondary,  but  related,  concern  of  the  PM  involves  the 
quality,  size,  and  growth  of  the  technical  staff.  The  work 
force  consists  of  high  and  low  quality  personnel  which  are 
assumed  distinguishable  to  the  PM  in  the  implementation  of 
worker  assignment  and  hiring/f iring  decisions.  As  in  the 
real  world,  limits  exist  on  the  availability  of  high  quality 
workers.  However,  it  is  assumed  that  the  PM  is  able  to  hire 
low  quality  (inexperienced)  personnel  as  necessary  to  fulfill 
labor  requirements.  Relative  to  inexperienced  personnel, 
high  quality  workers  are  assumed  to  have  higher  voluntary 
turnover  rates,  higher  salaries  and  higher  marginal  produc¬ 
tivity.  In  general,  the  DPM  simulates  the  project  manager 
(PM)  pursuing,  subject  to  various  operational  constraints  and 
pressures  (discussed  below) ,  the  attainment  of  personal 
backlog  and  staffing  goals  by  implementing  specific  work 
force  assignment  and  hiring/ firing  decisions  on  a  monthly 
basis . 

The  corporate-level,  on  the  other  hand,  is  concerned 
with  the  cash  flows,  cost  overruns  and  the  resultant  impact 
on  the  financial  position  of  the  firm.  If  the  quarterly 
performance  is  unsatisfactory,  pressure  is  applied  on  the 
project  manager  to  increase  cash  flow.  The  PM  is  able  to 
improve  cash  flow  by  increasing  the  amount  of  billable  time 
ckarged  against  existing  contracts  (i.e.  ,  increase  the  number 
of  workers  on  direct  activities)  and/or  decrease  current  cash 
expenses  (i.e.,  implement  firing  decisions).  An  additional 
concern  of  corporate  is  the  level  of  allocations  in  support 
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of  IR&D  and  administrative  expenses  levied  on  the  PM's 
organization. 

Finally,  the  DOD  monitors  the  performance  of  the  PM  with 
respect  to  individual  projects.  Specifically,  the  DOD  is 
concerned  with:  a)  total  costs  as  determined  by  cumulative 
billings  against  a  contract;  b)  performance  quality  as 
reflected  in  the  average  quality  of  personnel  used  in  satis¬ 
fying  the  SQMM  requirement;  and  c)  the  time  period  in  which 
the  product  is  available  relative  to  the  contracted  delivery 
date.  The  exhaustion  of  the  SQMM  balance  does  not  necessar¬ 
ily  coincide  with  that  of  the  dollar  budget  and,  dependent  on 
the  contract  type  utilized,  excess  costs  are  not  fully 
kiUsble  to  the  DOD.  When  quality  or  schedule  problems  are 
identified  by  the  DOD,  pressure  is  applied  on  the  PM.  If 

project  manager  pressure  does  not  result  in  improved  perfor- 

"  > 

mance,  the  DOD  may  convey  its  concerns  to  the  corporate- 
level,  dependent  on  the  severity  of  the  problem. 

In  summary,  the  interaction  of  the  various  organization¬ 
al  levels  and  their  associated  goals  and  expectations  com¬ 
bined  with  the  many  operational  factors  and  constraints  faced 
by  the  PM  provide  a  realistic  operating  environment  for  the 
DPM.  The  DPM  also  demonstrates  interesting  dynamic  proper¬ 
ties  in  that  PM  and  corporate-level  goals  are  modified 
throughout  the  simulation,  dependent  on  prolonged  success  or 
failure.  (DOD  goals  are  assumed  static  and  determined  with 
the  awarding  of  a  contract.)  Exhibit  1.3  illustrates  the 
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dynamic  interaction  of  the  PM,  corporate,  and  DOD 
organizational  levels  during  the  course  of  a  simulation. 


Exhibit  1.3 

1.3.  Model  Decomposition 

Exhibit  1.4  provides  a  listing  and  ranking  of  the 
various  pressures  and  goals  facing  the  PM  as  they  were 
incorporated  into  the  DPM  logic  flow.  The  paramount  concerns 
of  the  PM  involves  Corporate  Cash  Flow  Pressure,  DOD  Schedule 
Pressure,  and/or  Deficient  Quality  on  a  completed  proiect. 
The  PM  is  assumed  to  address  these  occurrences  before  consid¬ 
ering  remaining  problems.  Other  pressures  and  unsatisfied 
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goals  are  not  ignored,  but  are  given  secondary  consideration 
if  they  are  in  conflict  with  higher  ranking  priorities. 

The  next  level  of  influence  on  PM  behavior  is 
Corporate  Quality  Pressure  which  is  triggered  when  DOD 
quality  pressure  is  communicated  to  the  corporate-level,  and 
then  relayed  to  the  PM.  The  DOD,  as  mentioned,  approaches 
the  corporate— level  with  quality  concerns  only  if  previous 
DOD  Quality  Pressure  has  been  ineffective.  As  illustrated  in 
Exhibit  1.4,  this  may  occasionally  occur  as  the  Backlog  Goal 
of  the  project  manager  takes  precedence  over  DOD  Quality 
Pressure.  Finally,  the  Volume  Goal  of  the  project  manager  is 
the  lowest  ranking  consideration  in  the  attention  hierarchy. 


Determinants  of  Project  Manager  Behavior 


Level  1: 

Corporate  Cash  Flow  Pressure 

DOD  Schedule  Pressure 

Deficient  Quality  on  Completed  Project 

Level  2: 

Corporate  Quality  Pressure 

Level  3: 

Backlog  Goal 

Level  4: 

DOD  Quality  Pressure 

Level  5: 

Volume  (staff)  Goal 

Exhibit  1.4 

3.1.  PM  Manpower  Assianment  Sub-Model 

The  purpose  of  the  PM  Manpower  Assignment  Sub-Model 
is  to  assign  each  high  and  low  quality  worker  to  indi¬ 
rect  proposal  development  activities  or  to  a  specific 
project.  As  discussed,  the  PM  is  assumed  to  make 
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personnel  allocation  decisions  based  on  the  existence  of 
various  DOD  and  corporate  pressures,  the  achievement  of 
personal  goals  and  individual  constraints  due  to  backlog 
of  incomplete  projects. 

1.3.2.  Project  Update  Sub-Model 

Based  on  the  personnel  allocation  decision  made  by 
the  PM  with  respect  to  the  quantity  and  quality  of  work 
performed  on  each  project,  several  running  measures  are 
updated  for  each  project  worked  on.  Specifically,  the 
remaining  dollars  in  the  budget  and  the  required  SQMM 
balance  must  be  decreased  and  the  quality  index  of  work 
performed  adjusted  accordingly.  Finally,  the  time 
remaining  before  each  project  is  due  is  calculated. 

1.3.3.  Personnel  Sub-Model 

The  objective  of  the  Personnel  Sub-Model  is  to 
implement  the  decisions .  made  by  the  PM  with  respect  to 
alterations  in  the  size  of  the  work  force.  Labor  force 
attrition  is  also  reckoned  with  in  this  sub-model  and  is 
assumed  to  occur  simultaneously  with  the  PM's  hiring/ 
firing  decision  i.e.,  the  PM  is  unable  to  compensate 
for  attrition  in  the  current  period.  The  output  of  the 
Personnel  Sub-Model  is  a  description  of  the  end-of-the- 
period  labor  force  in  terms  of  size  and  quality 
composition . 

1-3.4.  Backlog  Determination  Sub-Model 

Based  on  the  output  of  the  previous  sub-models,  the 
Backlog  Determination  Sub-Model  calculates  the  number  of 


15 


months  of  backlog  (given  current  staffing  levels)  at  the 
end  of  the  period.  This  involves  determining  whether  a 
project  has  been  awarded  and,  if  so,  the  dollar,  deliv¬ 
ery  date  and  SQMM  size  of  the  new  project. 

1.3.5  Cash  Flow  Determination  Sub-Model 

The  Cash  Flow  Determination  Sub-Model  calculates 
the  direct  and  indirect  costs  associated  with  the  PM's 
operation  and  the  cumulative  billings  to  the  DOD  allowed 
for  the  direct  work  performed  on  individual  projects. 
Billings  against  a  particular  project  are  allowed  only 
if  the  budget  has  not  been  depleted.  Once  the  target 
cost  on  a  given  project  has  been  reached,  the  contrac¬ 
tually  specified  cost-sharing  formula  allocates  subse¬ 
quent  cost-overruns  between  the  contractor  and  the  DOD. 
If  the  ceiling  cost  is  exceeded,  all  further  expenses 
incurred  on  a  given  project  in  satisfaction  of  SQMM 
requirements  and/or  quality  standards  are  fully  absorbed 
by  the  contractor. 

1-3.6  Corporate  Goal  Adjustment  Sub-Model 

The  output  of  the  Corporate  Goal  Adjustment  Sub- 
Model  is  a  determination  of  the  level  of  appropriations 
in  support  of  corporate  administrative  and  IR&D  expenses 
that  is  levied  on  the  PM's  operation.  Also  a  decision 
as  to  whether  to  apply  cash  flow  pressure  on  the  PM  is 
calculated  in  this  sub-model. 
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1.3.7  PM  Goal  Adjustment  Sub-Model 

The  PM  Goal  Adjustment  Sub-Model  is  responsible  for 
modifying  the  PM's  personal  Backlog  and  Volume  goals 
based  on  the  success  of  past  and  current  performances 
and  the  existence  of  pressures  from  corporate  and  DOD 
*  levels. 

1.3.8  DOD  Sub-Model 

The  DOD  Sub-Model  monitors  progress  made  on  each  of 
the  projects  awarded  to  the  defense  contractor  and 
determines  whether  schedule  and/or  quality  pressure 
should  be  applied  at  the  PM  or  corporate  levels. 

1 • 4  Model  Solution 

All  sub-Mode Is  have  several  functional  categories  which 
collectively  describe  .the  work  flow  during  the  solution 
sequence.  Each  sub— model,  except  several  routines  in  the 
Corporate  Goal  Adjustment  Sub— Model  (which  are  solved  quar¬ 
terly)  ,  are  solved  each  time  period  of  the  simulation. 

As  shown  in  Exhibit  1.5,  the  simulation  begins  with 
solution  of  the  PM  Manpower  Assignment  Sub-Model.  This 
«.  involves  the  PM  review  of  the  operating  environment  (see 

Exhibit  1.3)  taking  note  of  various  pressures  and  goal 
.  The  labor  force  is  then  broken  down  between 
and  direct  activities  and  assigned  to  specific 
tasks;  either  proposal  writing  or  to  an  incomplete  project. 
Decisions  are  also  made  concerning  hiring  or  firing.  After 
solution  of  the  PM  Manpower  Assignment  Sub-Model,  each 
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project  in  the  backlog  is  updated  through  solution  of  the 
Project  Update  Sub-Model. 


Exhibit  1.5 

The  Personnel  Sub-Model  is  then  solved  which  determines 
the  size  and  quality  composition  of  the  labor  force  at  the 
end  of  the  period— —taking  into  account  the  hiring  and  firing 
decisions  of  the  project  manager  and  labor  force  attrition. 
The  Backlog  Determination  Sub-Model  utilizes  the  output  of 
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the  Personnel  Sub-Model  and  calculates  the  actual  months  of 
backlog  in  the  PM's  incomplete  project  inventory'  at  the  end 
of  the  period.  This  also  requires  determining  if  a  new 
project  has  been  awarded  in  the  current  period. 

After  solution  of  the  Cash  Flow  Determination  Sub-Model, 
a  the  Corporate  and  Project  Manager  Goal  Adjustment  Sub-Models 

are  solved.  Finally,  the  DOD  Sub-Model  which  monitors 
individual  project  performance,  is  solved. 

1.5.  Objectives  of  Contract  F33615-81-C-5034 

The  effort  under  contract  F33615-79-5151  was  devoted  to 
the  development  of  the  Multi  Project  DOD  Dependent  Firm 
Computerized  Decision  Process  Simulation  described  above. 
The  objective  of  contract  F33615-81-C-5034  was  to  validate 
the  DPM  simulation  and  its  application  to  the  development  and 
testing  of  alternative  incentive  schemes.  Specifically  the 
validation  phase  was  to  involve  scenario  analyses  (Section 
2) ,  sensitivity  analyses  (Section  3)  and  external  evaluations 
by  experienced  managers  (Section  4) .  It  was  also  intended  to 
demonstrate  the  use  of  the  simulation  as  a  management  tool 
..or  developing  and  evaluating  alternative  incentive  schemes. 
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SECTION  2:  SCENARIO  GENERATION  AND  ANALYSES 


2 . 1  Overview 

The  specific  objectives  of  the  scenario  Generation  and 
Analysis  phase  were:  a)  to  analyze  in  detail -the  simulated 
output  of  two  scenarios  as  plausibility  checks  of  DPM  consis-  a 

tency  and  reasonableness,  and  b)  to  demonstrate  how  the  DPM 
may  be  utilized  to  investigate  alternative  procurement 
policies  of  the  DOD  and  business  strategies  of  defense 
contractors .  In  the  scenario  analyses  the  DPM  is  solved  over 
72  decision  making  intervals  which  are  equivalent  to  six 
years  of  monthly  and  quarterly  decision  adjustments.  One 
year  or  12  periods  of  historical  data  is  necessary  to  ini- 
the  simulation.  The  database  was  designed  to  allow 
the  project  manager  (PM)  to  simultaneously  manage  a  maximum 
of  10  separate  projects  in  any  given  time  period.  No  con¬ 
straint  was  placed  on  the  number  of  projects  which  may  be 
awarded  over  the  72  month  horizon  of  the  simulation. 

2.1.1  Scenario  1:  Standard  Simulation 

Scenario  1  is  considered  the  standard  to  which 
other  simulations  are  compared.  The  assumption  was  made 
that  the  project  manager  is  awarded  a  Major  Project  in 
period  1  and  no  other  Major  Projects  are  obtained 
throughout  the  simulation.  No  mechanism  is  currently 
specified  which  translates  corporate  IR&D  allocations 
into  Major  Projects.  Because  the  project  manager  is  not 
awarded  another  Major  Project,  his  backlog  and  volume 
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goals  (as  well  as  associated  corporate  and  DOD  concerns) 
must  be  satisfied  by  the  attainment  of  new  spinoff 
projects  and  the  completion  of  the  initial  Major 
Project . 

2.1.2  Scenario  2;  No  DOD  Monitor 

The  purpose  of  generating  this  scenario  was  to 
investigate  the  effect  of  the  DOD  monitor  (and  the 
associated  quality  and  schedule  pressure  mechanism)  on 
the  behavior  of  the  project  manager  and  on  overall 
system  performance.  For  this  scenario  it  is  assumed 
that  the  DOD  will  not  apply  schedule  or  quality  pressure 
at  the  corporate  or  project  manager  level.  It  is 
further  assumed  that  the  DOD  will  not  accept  the  major 
project  unless  the  minimum  contractually  specified 
quality  goal  has  been  achieved.  Additional  high  quality 
work  is  required  if  the  quality  of  an  otherwise  complet¬ 
ed  project  is  less  than  the  minimum  acceptable  standard. 
As  in  Scenario  1,  only  one  Major  Project  is  assigned  to 
the  project  manager. 

2.1.3  Summary 

For  each  of  the  scenarios,  the  time  trends  of 
important  variables  were  analyzed  from  the  perspective 
of  the  project  manager,  corporate  management  and  the 
DOD.  The  results  which  are  reported  in  technical  report 
#1  support  the  reasonableness  of  the  DPM  in  month  13-72 
(the  first  12  months  of  each  simulation  run  are  needed 
for  initializing  the  DPM) . 
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2 . 2  POD  Policy  Experimentation 

This  phase  was  intended  to  demonstrate  how  the  DPM 
be  utilized  as  a  management  tool  in  the  formulation  and 
evaluation  of  alternative  procurement  policies  by  the  DOD. 

The  example  which  was  developed  as  part  of  the  contract  was 

for  illustrative  purposes  and  should  not  be  interpreted  as  an  4 

evaluation  of  existing  policies  or  recommendations  for 

.  *  * 
improvement . 

The  policy  experiment  was  based  on  a  set  of  DPM  scenari¬ 
os  that  were  generated  as  a  means  o  f  investigating  the 
effectiveness  (with  respect  to  project  performance)  of  two 
policy  instruments.  Both  instruments,  or  control  variables, 
are  contractual  specifications  dealing  with  the  allocation  of 
project  costs.  The  first  involves  the  "share"  formula  which 
determines  the  percent  of  actual  costs  greater  than  target- 
b  ut  less  than  ceiling — that  must  be  absorbed  by  the  contrac¬ 
tor  (i.e.,  non-reimbursable  cost  overruns).  Most  attention 
in  the  literature  has  been  given  to  the  optimal  specification 
of  the  sharing  proportion  or  "share".  The  second  policy 
instrument  analyzed  concerns  the  setting  of  ceiling  relative 
to  target  costs  or  the  "range"  of  ceiling  cost  over  target 
cost.  * 


★ 

This  section  and  2.3  were  derived  from  Sections  1.4  and  1.5 
of  Technical  Report  #1,  pp.  28-38. 
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A  serious  problem  associated  with  standard  models  of 
incentive  contracting  is  a  computational  limitation  of 
concurrently  evaluate  multiple  DOD  goals  (e.g. ,  cost,  sched¬ 
ule,  quality)  and  policy  instruments  (e.g.,  target  and 
ceiling  costs,  sharing  proportion,  progress  payments  and 
project  award  algorithm) .  The  DPM  methodology  is  particular¬ 
ly  well-suited  to  deal  with  this  common  deficiency. 

2.2.1  Multiple-Scenario  Generation 

In  order  to  evaluate  the  effectiveness  of  a  given 
policy,  or  combination  of  policies,  it  is  necessary  to: 
a)  determine  that  the  simulation  model  is  a  reasonably 
accurate  representation  of  firm  behavior  based  on 
observations  of  past  performance,  DOD  policies  and 
contractor  operating  strategies,  b)  identify  and  define 
performance  output  variables  of  interest  to  policy 
makers,  and  c)  assign  a  feasible  range  of  experimental 
values  to  the  policy  instrument (s)  under  consideration. 
The  overall  concern  of  this  research  project  is  focused 
on  the  first  criteria.  This  section  addresses  criteria 
b  and  c . 

The  following  performance  indicators  are  evaluated 
over  a  5  year  simulation  horizon: 

Production  Volume  -  the  total  volume  of  work 

performed  as  reflected  in 
the  cumulative  SQMM's 
(Standard  Quality  Work, 
Month  of  Work)  of  all 
completed  projects, 
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Performance  Quality 


the  average  quality  of 
completed  projects  weight¬ 
ed  by  their  respective 
SQMM  size. 


Performance  Efficiency 


the  average  cost  to  the 
DOD  per  completed  SQMM  and 
the  total  cost  (including 
the  contractor's  share) 
per  SQMM, 


Cost  Control  Efficiency 


the  average  cost  overrun 
per  project  as  a  percent 
of  target  costs,  and. 


Schedule  Performance 


the  average  number  of 
months  late  per  completed 
project. 


As  mentioned  above,  two  policy  instruments  are 
being  evaluated:  the  contractor's  share  of  cost  over¬ 
runs  and  the  range  of  ceiling  over  target  cost.  The 
basic  Standard  Scenario  is  solved  over  72  periods  (12 
time  periods  are  required  for  initialization)  for  each 
possible  combination  of "values  assigned  to  the  two 
policy  instruments.  Three  values — low,  medium  and 
high — have  been  chosen  for  each  instrument:  15%,  40% 

and  75%,  respectively,  for  the  sharing  proportion  and 
110%,  125%  and  150%,  respectively,  for  the  range.  The 
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low  values  are  typical  of  prevalent  incentive  contracts. 
The  medium  and  high  values  have  been  chosen  to  represent 
more  extreme  options. 

2.2.2  Policy  Analysis 

Table  2.1  displays  the  results  of  the  policy 
experiments  with  respect  to  the  volume  of  work  (SQMM's) 
performed.  (The  effect  of  a  given  policy  on  performance 
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outcome,  averaged  over  all  values  of  the  other  policy 
instrument,  is  found  at  the  end  of  the  appropriate  row 
or  column.)  There  appears  to  be,  for  any  given  share,  a 
strong  tendency  for  higher  ceilings  to  be  associated 
with  increased  output.  The  average  output  of  3.299 
SQMM  s  with  a  low  ceiling  (110%  of  target  costs)  com¬ 
pared  with  the  significantly  higher  output  of  3,920 
SQMM ' s  associated  with  a  high  ceiling  (150%  of  target 
costs)  suggests  that  all  else  egual,  contractors  inter¬ 
pret  high  ceilings  as  inducements  to  expand  internal 
capacity  (workforce) .  This  tendency  is  also  observed, 
to  a  lesser  extent,  in  the  effects  of  alternative 
sharing  propositions:  high  shares  (75%)  result  in  an 
average  output  of  3.515  SQMM's  while  low  shares  (15% 
result  in  3,602  SQMM's  being  completed).  Thus,  for 
policy  makers  primarily  concerned  with  promoting  high 
levels  of  production,  a  low  sharing  proportion  combined 
with  high  ceiling  costs  appears  to  be  appropriate. 
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Table  2.1  Production  Volume  (SQMM's) 


Ceiling  Cost  as  % 
of  Target  Cost 


Contractor's 
Share  of  Cost 
Overruns 


15% 

40% 

75% 


110% 

125% 

150% 

3,129 

3,542 

4,046 

3,278 

3,674 

3,903 

3,400 

3,334 

3,811  | 

3,299 

3,517 

3,920 

3,602 

3,618 

3,515 


High  ceilings  and  low  shares,  however,  are  associ¬ 
ated  with  lower  average  quality  performance.  As  shown 
in  Table  2.2  the  average  effect  of  a  high  ceiling  over 
all  sharing  proportions  on  performance  quality  is  .840 
while  low  ceilings  result  in  an  average  quality  of  .848. 
Likewise,  the  average  effect  on  performance  quality  of  a 
low  sharing  proportion  is  .84  while  a  high  share  is 
associated  with  an  average  performance  quality  of  .846. 
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Table  2 . 2  Performance  Quality 


Ceiling  Cost  as  % 
of  Target  Cost 


Contractor1 s 
Share  of  Cost 
Overruns 


110%  125%  150% 

15% 

40% 

75% 

. 843  . 841  . 840 


.847 

.834 

.840 

.848 

.840 

.839 

.348 

.850 

.840 

.340 

.842 

.846 


These  results  suggest  that  a  dichotomy  exists 
between  policies  which  promote  high  production  volume 
and  those  which  increase  performance  quality.  In  the 
first  case,  low  sharing  proportions  combined  with  high 
ceilings  incentivize  the  contractor  to  increase  capacity 
because  a  large  amount  (50%  of  target  costs)  of  low  cost 
(15%  share)  funds  become  available  to  support  workforce 
expansion.  Rapid  expansion  of  the  workforce,  however, 
decreases  the  quality  of  project  performance  due  to  the 
relative  scarcity  of  high  quality  personnel.  In  addi¬ 
tion,  the  project  manager  confronted  with  a  larger 


27 


workforce  utilizes  a  significant  proportion  of  high 
quality  personnel  on  proposal  writing  (rather  than  on 
direct  production  activities)  due  to  the  increased 
difficulties  of  maintaining  an  adequate  backlog. 

Table  2.3  provides  a  cost  efficiency  ranking  of  the 
various  range/ share  policy  combinations  from  the  per¬ 
spective  of  the  DOD.  As  the  results  indicate,  DOD  cost 
per  SQMM  is  not  significantly  influenced  by  alterations 
in  the  ceiling  cost.  As  expected,  high  sharing  propor¬ 
tions  have  a  significant  negative  impact  on  DOD  average 
costs  per  SQMM:  $6.01  and  $5.77  for  low  and  high 

sharing  proportions,  respectively.  More  importantly, 
from  the  perspective  of  social  efficiency,  the  total 
actual  costs  of  development  (including  the  contractor's 
share)  as  displayed  in  Table  2.4  is  also  decreased  as 
the  sharing  proportion  is  increased.  The  average  cost 
per  SQMM  over  all  ceiling  values  decreases  from  $6,079 
to  $.065  as  the  share  is  increased  from  15%  to  75%.  In 
addition,  average  total  costs  increase  ($6,054  to 
$6,090)  as  the  ceiling  cost  is  increased  from  110%  to 

v. 

150%. 
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Table  2 . 3  POD  Cost  Per  SQMM  ($000) 


Ceiling  Cost  as  % 
of  Target  Cost 


1102  125%  150% 

t 

15%  , 

Contractor's 
Share  of  Cost  40% 

Overruns 

75% 


5.89  5.89  5.90 

Table  2.4  Total  Costs  Per  SQMM  ($000) 


5.992 

6.020 

6.022 

5.887 

5.898 

5.912 

5.779 

5.765 

5.764 

6.01 


5.90 


5.77 


Ceiling  Cost  as  % 
of  Target  Cost 


Contractor's 
Share  of  Cost 
Overruns 


110% 

125% 

150% 

15% 

6.055 

6.092 

6.091 

4  0% 

6.052 

6.079 

6.090 

75% 

6.054 

6.061 

6.080 

6.079 

6.074 

6.065 


6.054 
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6.077 


6.090 


The  results  presented  in  Tables  2.3  and  2.4  suggest 
that  the  sharing  proportion  may  be  a  potentially  useful 
policy  instrument  to  control  actual  development 
costs — but  only  when  sharing  proportions  approach  levels 
of  40-75%.  When  sharing  proportions  are  applied  at  low 
levels  (see  Table  2.4,  15-40%),  very  little  cost  im¬ 

provement  results.  This  is  a  direct  result  of  the 
previously  mentioned  capacity  and  quality  implications 
of  low  sharing  proportions.  High  sharing  proportions  in 
the  40-75%  range  are  shown  to  result  in  improved  cost 
efficiency.  The  project  manager  faced  with  high  sharing 
proportions  (and  to  a  lesser  extent,  low  ceiling  costs) 
is  forced  to  run  a  "tight  ship"— i.e.,  no  unnecessary 
expansion  in  staff  and/or  overhead  expenses. 

Table  2.5  Cost  Overruns  as  %  of  Target  Costs 

Ceiling  Cost  as  X 
of  Target  Cost 

110%  125%  IStt 

15% 

Contractor1 s 
Share  of  Cost  40% 

Overruns 

75% 


7. S3 


7.41 

8.43 

8.12 

7.27 

8.06 

7.93 

6.45 

7.16 

7.45 

7.S9 


7.75 


7.02 


7.04 


7.88 


The  results  of  Table  2.5,  display  cost  overruns  as 
®  P®^cent  of  target  cost  for  each  of  the  range/ share 
scenarios  and  are  consistent  with  the  results  of  Tables 
2.3  and  2.4.  Lower  cost  overruns  are  consistently 
associated  with  higher  sharing  proportions  for  any  given 
ceiling.  Again,  it  is  important  to  note  that  a  much 
greater  effect  is  found  with  high  sharing  proportions 
rather  than  low. 

Finally,  Table  2.6  shows  the  average  number  of 
months  late  per  completed  project.  While  the  sharing 
proportion  does  not  show  a  directional  effect  on  sched¬ 
ule  delays,  low  ceilings  (110—125%)  seem  to  be  signifi¬ 
cantly  associated  with  schedule  delays.  The  average 
overall  sharing  associated  with  a  low  ceiling  is  3.26 
months..  With  a  high  ceiling  the  average  number  of 
months  late  drops  to  .8.  This  association  is  consistent 
with  the  results  shown  in  Table  2.1  for  production 
volumes.  Increased  internal  capacities  due  to  low 
shares  and  high  ceilings  increase  he  likelihood  that 
projects  will  be  completed  on  time. 
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Table  2.6  Average  Number  of  Months  Late  Per  Project 


Ceiling  Cost  as  X 
of  Target  Cost 


Contractor1 s 
Share  of  Cost 
Overruns 


15% 


40% 


753 


uox 

125% 

150X 

3.23 

4.57 

0 

2.92 

3.43 

0 

3.54 

2.08 

2.4 

3.26 

3.36 

.8 

2. 60 

2.12 

2.71 


2 . 3  Biased  Cost  Estimates  and  Project  Performance 

Much  controversy  has  traditionally  been  associated  with 
cost  overruns  and  their  viability  as  accurate  indicators  of 
contractor  efficiency.  It  is  generally  accepted  that  biased 
cost  estimates  are  an  important  factor  in  explaining  cost 
overruns  (i.e.,  "low-balling",  "buying-in",  etc.).  The 
objective  of  this  section  is  to  demonstrate  how  the  DPM  may 
be  utilized  to  investigate  the  multi-dimensional  effect  on 
project  performance  of  biased  cost  estimating  behavior. 

The  current  experiment  requires  solving  the  DPM  under 
the  identical  operating  conditions  (e.g.,  sharing  proportion. 
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cost/  stc . )  of  the  Standard  Scenario  except  that  he 
initial  estimated  target  cost  is  25%  below  the  previous 
estimated  amount.  (It  will  be  recalled  that  the  Standard 
Scenario  involves  a  sharing  proportion  of  15%  and  a  ceiling 
equal  to  125%  of  target  cost.)  Table  2.7  presents  the 
results  of  the  2  scenarios  under  consideration. 

Table  2.7  Accurate  vs.  Biased  Cost  Estimates 


Output  Variable  Standard 

1.  Production  Volume  3.542 

2.  Performance  Quality  .834 

3.  DOD  Cost/SQMM  6.02 

4.  Total  Cost/SQMM  6.09 

5.  Cost  Control  Efficiency  8.43 

6.  Schedule. Performance  4.57 


Biased  Estimate 

3.599 

.  846 
5.85 
6.07 
13.1 
1.07 


A  cursory  examination  of  the  data  suggests  that  biased 
estimates  do,  in  fact,  influence  project  performance  because 
all  but  2  of  the  output  variables  (production  volume  and 
total  cost  per  SQMM)  are  significantly  different.  A  somewhat 
surprising  result,  at  first  glance,  is  that  the  performance 
quality,  DOD  cost  per  SQMM  and  schedule  performance  indices 
are  imp  roved  when  low  initial  cost  estimates  on  the  Major 
Project  are  submitted  by  the  contractor.  Relative  to  the 
Standard  Scenario:  quality  has  improved  from  .834  to  .846, 
DOD  cost  per  SQMM  has  been  reduced  from  $6.02  to  $5.85  and 
the  average  month's  late  per  completed  project  has  decreased 
from  4.57  to  1.07.  As  expected,  cost  control  efficiency  is 
significantly  worse  when  low  estimates  are  submitted. 
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As  originally  hypothesized,  the  project  manager  in  the 
DPM  is  assumed  to  pursue  operational  goals  concerning  backlog 
iflternal  capacity  (staff)  in  a  fairly  autonomous  fashion 
as  long  as  corporate  and  DOD  goals  are  satisfied.  When 
P^^klsms  arise  (i.e.,  unsatisfied  goals) ,  conflict  between 
decision  centers  occurs  and  project  manager  goals  may  be  „ 

compromised  dependent  on  the  source  and  severity  of  the 
resultant  pressure  applied  to  the  project  manager. 

A  low  target  cost  on  the  initial  Major  Project  will,  all 
else  equal,  result  in  the  Project  Manager's  organization 
having  a  less  favorable  cash  flow  performance — especially  as 
the  SQMM  requirement  of  the  Major  Project  is  nearing  comple¬ 
tion  (time  periods  40-45,  approximately)  .  This  situation, 
the  resultant  Cash  Flow  Pressure  from  the  corporate 
level,  will  effectively  prevent  a  significant  expansion  in 
staff  normally  associated  with  large  contract  awards.  In 
addition,  with  a  smaller  monthly  payroll  fewer  newly  awarded 
projects  (and  high  quality  personnel  on  proposal  writing 
activities)  are  required  in  order  for  the  Project  Manager  to 
maintain  backlog  within  acceptable  boundaries.  In  short,  low 

* 

initial  cost  estimates  on  the  Major  Project  prevent  rapid 
expansion  in  internal  capacity  and  the  associated  problems  of  * 

a  lower  quality  workforce  and  larger  payroll  (direct  and 
overhead)  obligations.  Although  short-run,  operational 
problems  may  plague  the  contractor  (e.g.,  vulnerable  cash 
position,  schedule  slippages,  employee  morale  and  attrition) 
the  long-run  (5-year)  effect  on  overall  project  performance 
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with  respect  to  quality,  production  efficiency  and  delivery 
is  improved. 

2 . 4  Summary 

Scenario  analyses  was  utilized  to  explore  the  general 
behavioral  pattern  of  the  DPM  and  demonstrate  how  the  DPM  may 
be  used  to  evaluate  alternative  procurement  policies  and 
investigate  other  problem  areas  of  concern. 

The  next  phase  of  the  research  involved  a  sensitivity 
analysis  of  the  model  by  using  regression  techniques  to 
examine  the  underlying  determinants  of  model  behavior.  The 
DPM's  many  policy  instruments,  initial  values  and  behavioral 
adjustment  parameters  are  tested  for  their  effect  on  impor¬ 
tant  performance  outcomes  to  determine  whether  the  model 
application  to  real-world  phenomena  can  be  considered  viable. 
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SECTION  3:  SENSITIVITY  ANALYSIS* 


3 . 1  Overview 

The  objective  of  this  phase  was  to  investigate  the 
underlying  properties  of  the  DPM  in -more  detail.  The  proce¬ 
dures  chosen  to  investigate  the  relationship  between  the 
DPM's  exogenous  factors  (i.e.,  policy  instruments,  initial 
conditions  and  adjustment  parameters)  and  endogenous  respons¬ 
es  (i.e.,  performance/behavior  indices)  was  multiple  regres¬ 
sion  analysis.  It  is,  of  course,  impractical  to  attempt  to 
investigate  all  factors  of  possible  consequence  due  to  the 
complexity  of  the  DPM.  The  non-policy  factors  (i.e.,  initial 
conditions  and  behavioral  adjustment  parameters)  that  were 
investigated  for  significance  were  chosen  based  on  a  priori 
judgments  of  possible  operational  importance.  However,  all 
policy  instruments  currently  specified  (and  identified)  in 
the  DPM  were  included  in  the  sensitivity  analysis. 

Because  of  the  many  real-world  intricacies  of  the  system 
under  study ,  the  simulation  model  is  likely  to  contain 
complexities  which  disguise  important  variable  interactions 
which  may  appear  intuitive  and  obvious,  after  analyzing 
simulated  performance  data,  but  are  nevertheless  difficult  to 
comprehend  beforehand.  It  is  necessary,  therefore,  to  use  a 
good  deal  of  a  posteriori  reasoning  in  order  to  adequately 
investigate  the  behavioral  properties  of  the  DPM. 


★ 

This  section  is  based  on  Technical  Report  #2. 
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This  is  not  meant  to  imply  that  none  of  the  variable 
interactions  evaluated  as  part  of  the  sensitivity  analysis 
hsve  predictable  relationships.  To  the  contrary,  many 
variables  which  were  tested  have  predictable  directional 
effects  on  specific  performance  indices.  These  straightfor— 
d  ward  associations  are  a  direct  result  of  the  DPM's  structure 

and  internal  specification.  Furthermore,  a  good  many  of 
these  intuitive  relationships  have  been  identified  and 
analyzed  in  the  course  of  describing  various  scenarios.  For 
example,  a  workforce  consisting  of  higher  quality  personnel 
is  expected  to  contribute,  all  else  equal,  to  higher  perfor— 
quality.  For  the  most  part,  however,  the  analysis 
focused  on  underlying  properties  which,  while  perhaps  less 
intuitive,  are  equally  important  in  understanding  the  opera¬ 
tional  behavior  of  the  DPM.  More  specifically,  behavioral 
properties  of  the  DPM  were  investigated  which  may  very  well 
exist  in  the  real  world  but  are  extremely  difficult  to 
comprehend  in  a  non-controlled  environment. 

3 . 2  Basic  Procedure  of  Analysis 

The  methodology  used  to  investigate  the  sensitivity  of 
DPM  behavior  to  various  internal  factors  is  straightforward. 
The  first  step  was  to  define  key  output/performance  measures 
from  the  perspective  of  the  DOD,  Project  Manager  and  Corpo¬ 
rate  Top  Management.  In  addition,  the  appropriate  time 
periods  in  which  to  observe  their  values  must  be  identified. 
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The  following  performance  variables  were  considered: 


XlDODt 

X2D0Dt 

X3D0Dt 

X4D0Dt 

X5D0Dt 

YlPM 

Y2PMt 

Y3PMt 

ZlCORPt 

Z2C0RPt 


DOD  schedule  performance  measure  defined  as 
the .  cumulative  number  of  months  late  on  all 
projects  completed  during  the  relevant  time 
period  divided  by  the  number  of  completed 
projects, 

DOD  cost  control  performance  measure  defined 
as  the  cumulative  cost  overruns  on  all  pro¬ 
jects  completed  during  the  relevant  time 
period  divided  by  the  cumulative  target  costs, 

DOD  quality  performance  measure  defined  as  the 
average  quality  of  all  projects  completed 
during  the  relevant  time  period  weighted  by 
their  respective  SQMM  sizes, 

DOD  cost  efficiency  performance  measure 
defined  as  the  cumulative  dollars  billed  on 
all  completed,  projects  during  the  relevant 
time  period  divided  by  their  cumulative  SQMM 
sizes, 

social  cost  efficiency  performance  measure 
defined  as  the  cumulative  actual  costs  in¬ 
curred  on  all  completed  projects  during  the 
relevant  time  period  divided  by  their  cumula¬ 
tive  SQMM  sizes, 

Project  Manager  labor  force  volume  measure 
defined  as  the  average  monthly  size  of  the 
labor  force  over  the  relevant  time  period. 

Project  Manager  backlog  (security)  measure 
defined  as  the  average  monthly  size  of  the 
backlog  over  the  relevant  time  period. 

Project  Manager  labor  force  quality  measure 
defined  as  the  average  quality  of  the  labor 
force  over  the  relevant  time  period. 

Corporate  level  cash  flow  performance  measure 
defined  as  the  cumulative  net  cash  flows  over 
the  relevant  time  period,  and 

Corporate  level  profitability  measure'  defined 
as  the  cumulative  net  cash  flows  over  the 
relevant  time  period  divided  by  the  cumulative 
billings  to  the  DOD. 
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Ths  DPM  was  solved  over  a  time  horizon  of  72  periods  or 
5  years  of  monthly  decision  adjustments.  (The  reader  will 
recall  that  12  periods  of  historical  data  are  required  for 
initialization.)  Each  of  the  100  quasi-observations  which 
comprise  the  experimental  database  required  a  full  solution 
of  the  DPM.  Four  time  periods  are  considered  for  analysis: 
years  1-5,  2-5,  2-3,  and  4-5.  Years  1-5  are  evaluated  in 

order  to  capture  the  aggregate  effect  of  variations  in 
internal  factors  on  the  output  variables  defined  above. 
However,  in  order  to  somewhat  dampen  the  start-up  effect  of 
initial  conditions  on  performance  indicators,  years  2-5  were 
chosen  for  more  detailed  examination.  Finally,  the  2-5  year 
time  span  is  divided  in  order  to  investigate  DPM  behavior 
during  and  after  completion  of  the  Major  Project  —  which 
typically  is  completed  in  year  3. 

3.2.1  Initial  Conditions 

Of  the  eight  initial  conditions  analyzed,  two  were 
identified  as  having  a  fairly  consistent  influence  on 
performance  outcomes:  the  size  of  the  Project  Manager's 
workforce  in  the  initial  period  of  the  simulation 
(LABOR)  and  the  propotion  of  the  initial  workforce 
considered  high  quality  (PH) .  As  expected,  larger 
capacity  (staff)  and  higher  quality  personnel  result  in 
improved  cost  control  performance.  Social  cost  effi- 
ciency  (total  cost  per  SQMM)  is  also  shown  to  improve 
with  increases  in  the  total  workforce  and  the  initial 
proportion  considered  high  quality,  although  neither 
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variable  significantly  influences  the  cost  to  the  DOD 
per  SQMM.  Finally,  a  higher  quality  initial  workforce 
has  the  predictable  effect  on  performance  quality. 

With  respect  to  performance  variables  of  interest 
to  the  Project  Manager,  high  initial  workforce  quality 
facilitates  the  achievement  of  labor  force  goals  and 
quality  objectives  as  well  as  higher  average  backlogs. 
In  each  case,  the  directional  effect  of  significant 
initial  conditions  on  Project  Manager  performance 
variables  are  consistent  with  our  intuition. 

An  interesting  result  is  that  cumulative  cash  flow 
performance  is  favorably  influenced  by  increases  in  the 
size  of  the  initial  workforce.  This  is  probably  the 
result  of  the  Project  Manager  having  more  billable  time 
available  to  be  charged  against  initial  contracts  and 
not  having  to  hire  additional  workers.  Initial  backlog 
goals  also  positively  influence  cash  flow  performance. 

A  Project  Manager  with  greater  concerns  for  maintaining 
high  levels  of  backlog  will  be  in  a  better  position  to 
afford  extended  periods  of  negative  cash  flow  (or  at 
least  decrease  their  severity) . 

It  should  be  mentioned  that  the  significance  of 
several  initial  conditions  discussed  above  to  various 
DPM  performance  indices  is  not  unexpected  or  undesir¬ 
able.  In  the  first  place,  extremely  large  ranges  of 
possible  values  were  tested:  the  surprising  result 

would  have  been  if  they  were  found  to  be  inconsequential 
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to  DPM  performance.  Secondly,  the  observed  effects  of 
variations  in  initial  conditions  can  play  an  important 
role  in  validating  the  DPM  by  providing  a  plausibility 
check  on  reasonableness  and  consistency.  For  example, 
if  significant  increases  in  the  initial  endowment  of 
high  quality  personnel  had  no  effect  whatsoever  on 
performance  outcomes,  a  re-evaluation  of  the  DPM's 
internal  specifications  would  be  required. 

3.2.2  Behavioral  Adjustment  Parameters 

None  of  the  behavioral  parameters  were  significant 
at  the  5%  level  except  in  one  case.  The  log  linear 
regression  suggests  that  the  average  size  of  the  Project 
Manager's  backlog  is  negatively  related  to  the  rate  at 
which  workers  are  transferred  from  indirect  to  direct 
activities.  The  explanation  for  this  relationship  is 
sslf— evident  once  consideration  is  given  to  the  mecha¬ 
nism  by  which  the  Project  Manager  controls  the  level  of 
backlog. 

Projects  awarded  by  the  DOD  are  assumed  to  be 
instigated  by  the  submission  of  proposals  prepared  by 
personnel  classified  as  indirect.  The  rate  at  which  new 
proposals  are  generated  is  a  function  of  the  size  of  the 
indirect  workforce.  The  higher  the  specific  rate  at 
which  workers  are  transferred  (when  required)  to  direct 
activities,  the  fewer  proposals  are  submitted  and,  the 
fewer  projects  awarded  by  the  DOD. 
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3.2.3  POD  Policy  Instruments 

On  the  13  DOD  policy  instruments  investigated,  2 
are  clearly  influential  on  DPM  performance  in  that 
variations  in  their  values  are  significantly  associated 
(at  the  5%  level)  with  at  least  50%  of  the  performance 
variables.  The  weight  assigned  by  the  DOD  to  past 
contractor  performance  when  assessing  the  attractiveness 
of  a  newly  submitted  proposals  (henceforth  referred  to 
as  the  capture  rate  algorithm)  is  significantly  associ¬ 
ated  with  all  output  variables  except  DOD "quality  and 
cost  efficiency  performance.  The  higher  the  weights 
assigned  to  past  project  performance  (i.e.,  mean  and 
variance  of  past  quality  indices)  the  lower  are  cost  in 
the  long  run  and  social-  efficiency.  Higher  weights  on 
past  performance  (and  in  this  case  also  to  the  weight 
assigned  to  proposal  quality)  are’ also  associated  with 
improved  Project  Manager  staffing  and  backlog  perfor¬ 
mances,  but  are  negatively  associated  with  the  average 
quality  of  the  workforce.  This  latter  effect  is  a 
direct  result  of  the  large  expansion  in  staff  and  the 
relative  scarcity  of  high  quality  personnel.  Finally, 
the  weight  assigned  to  past  performance  is  shown  to  be 
positively  associated  with  total  cash  flow  performance 
and  total  cash  flow  as  a  percent  of  total  billings. 

The  other  policy  instrument  identified  as  having  a 
significant  influence  on  overall  DPM  performance  is  the 
scale  parameter  which  adjusts  (contractually  specified) 


42 


target  costs  to  an  amount  greater  than  or  less  than 
estimated  costs.  Higher  target  costs  are  shown  to 
improve  schedule  performance,  cost  control,  and  project 
performance  quality.  Conversely,  low  target  costs  are 
associated  with  schedule  delays,  cost  overruns  and 
quality  problems. 

It  is  clear  that  higher  target  costs  will,  ceteris 
paribus,  result  in  smaller  cost  overruns.  Similarly, 
increasing  target  cost  is  synonymous  with  increasing 
cash  flow  potential  and  man— months  of  backlog  associated 
with  any  given  contract.  By  implicitly  reducing  the 
demands  made  on  the  Project  Manager  to  win  new  contract 
awards,  heretofore  unutilized  resources  become  available 
for  direct  project  work — i.e.,  the  size  and  quality  of 
personnel  assigned  to  writing  new  proposals  is  reduced. 
This,  in  turn,  has  the  expected  effect  on  project 
schedule  and  quality  performance  measures.  It  is 
interesting  to  note  that  increasing  target  cost  also 
increases  the  size  of  the  workforce  and  average  level  of 
backlog,  but  does  not  significantly  affect  cash  flow 
performance . 

Several  other  policy  variables  also  had  a  signifi¬ 
cant  influence  on  project  performance  outcomes.  For 
example,  increases  in  the  quality  goal  on  the  Major 
P^ojsct  and  the  magnitude  of  the  new  proposals  capture 
rate  worsens  schedule  performance.  Increases  in  the 
negotiated  overhead  rate  worsens  DOD  cost  and  total  cost 
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performance.  Increases  in  the  contractor's  share  of 
cost  overruns  and  increases  in  the  contractor's  fee  are 
both  shown  to  improve  cost  control  performance.  Final¬ 
ly,  increasing  the  overhead  rate  improves,  as  expected, 
cumulative  cash  flow  performance. 

3 • 3  Analysis  of  Endogenous  Performance  Variables 

The  objective  of  this  section  is  to  discuss  in  more 
detail  the  results  of  the  sensitivity  analysis  with  respect 
to  various  endogenous  performance  variable.  The  focus  is  on 
the  sensitivity  of  the  various  performance  indices  to  policy 
instruments.  In  addition,  emphasis  was  given  to  long-run 
effects  (i.e.,  overall  and/or  later  periods  of  the  simulation 
as  opposed  to  immediate  short-term  effects)  due  to  the  desire 
to  minimize  initial  condition  effects. 

3.3.1  Schedule  Performance 

High  target  costs  and  low  weights  assigned  to 
overall  past  performance  and  to  proposal  quality  were 
shown  to  improve  schedule  performance.  Similarly, 
higher  minimum  levels  of  acceptable  quality  performance 
the  Major  Project  worsen  overall  schedule  performance 
in  years  2—5  but  the  effect  is  reduced  in  the  later 
years  of  the  simulation.  This  is  expected  because  the 
Major  Project  was  usually  completed  by  the  third  or 
fourth  years  of  the  simulation.  Higher  negotiated 
overhead  rates  were  shown  to  be  associated  with  improved 
performance;  in  this  case  the  effect  becomes  more 
pronounced  in  the  later  years  after  expiration  of  the 
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Major  Project.  This  is  consistent  with  our  intuition 
that  higher  overhead  rates,  ceteris  paribus,  increases 
internal  capacity  and  subsequent  project  performance 
indices . 

3.3.2  Cost  Control  Performance 

As  discussed  in  the  previous  section,  a  larger  and 
higher  quality  workforce  can  be  expected  to  reduce  cost 
overruns .  This  is  confirmed  in  all  three  periods  under 
analysis.  Similarly,  higher  target  costs,  ceteris 
paribus  result  in  smaller  cost  overruns.  Finally,  the 
long-run  (years  4-5)  effect  of  the  sharing  proportion  on 
cost  control  should  be  mentioned.  Increasing  the 
sharing  proportion  results  in  improved  cost  control — but 
not  until  the  late  periods  of  the  simulation  when  the 
Major  Project  is  completed.  This  results  from  not 
specifying  intermediate  cost  and  performance  milestones 
for  the  major  project. 

3.3.3  Quality  of  Performance 

The  prominent  determinants  of  performance  are  the 
proposition  of  the  initial  workforce  considered  high 
quality  and  the  relationship  of  target  costs  to  estimat¬ 
ed  costs.  As  expected,  increasing  the  quality  of  the 
workforce  allows  the  Project  Manager  to  improve  project 
performance  quality. 

Increases  in  target  costs  are  shown  to  influence 
performance  quality:  the  higher  the  target  costs, 

ceteris  paribus,  the  higher  the  resultant  quality.  In 
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the  later  periods,  higher  maximum  backlog  goals  are 
associated  with  lower  guality.  An  explanation  for  this 
letter  influence  is  that  a  Project  Manager  concerned 
with  maintaining  backlog  and  securing  new  projects 
(i.e.,  growth)  is  likely  to  utilize  high  guality  person¬ 
nel  on  indirect  (proposal  writing)  activities  rather 
than  direct. 

3.3.4  POD  Cost  Efficiency 

The  regression  results  suggest  that  the  costs  per 
SQMM  are  strongly  influenced  by  the  negotiated  overhead 
rate.  Higher  overhead  rates  are  consistently  associated 
wi'th  higher  costs.  In  addition,  the  higher  the  weight 
assigned  to  past  performance  guality  in  the  capture  rate 
calculation  the  lower  are  costs  in  the  long-run. 

3.3.5  Social  Cost  Efficiency 

The  sensitivity  analysis  corresponding  to  total 
social  costs  per  SQMM  (i.e.,  the  amount  billed  to  the 
DOD  as  well  as  the  cost  overruns,  if  any,  absorbed  by 
the  contractor)  indicated  that  the  overhead  rate  plays  a 
strong  role  in  determining  total  costs  per  SQMM. 
However,  social  cost  efficiency  is  influenced  by  initial 
policy  instruments  including  the  weight  assigned  to 
overall  (average  and  variance  of  guality)  past  perfor¬ 
mance,  the  range  of  ceiling  over  target  costs,  and  to  a 
lesser  extent,  the  contractor's  share  of  cost  overruns. 
An  interesting  result  is  that,  all  else  egual,  higher 
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ceiling  costs  and  sharing  proportions,  improve  social 
efficiency . 

3.3.6  Project  Manager  Considerations 

The  operational  concerns  of  the  Project  Manager: 
the  average  size  of  the  workforce,  level  of  backlog  and 
quality  of  personnel  were  shown  to  be  associated  with 
the  capture  rate  on  new  proposals  and  are  generally 
significant  predictors  of  Project  Manager  performance 
concerns  as  to  the  target  cost  relative  to  estimated 
costs . 

The  positive  influence  of  the  new  proposals  capture 
rate  on  the  average  size  of  the  workforce  was  expected 
because  a  higher  capture  rate,  all  else  equal,  will 
result  in  more  projects  being  awarded  and  a  greater 
opportunity  for  the  Project  Manager  to  expand  the  size 
of  the  staff.  Likewise,  increasing  the  range  of  ceiling 
over  target  costs  was  expected  to  provide  inducements 
for  expansion  of  staff. 

A  somewhat  puzzling  result  at  first  glance  is  the 
significant  short-run  effect  of  high  Major  Project 
quality  standards  on  labor  force  size.  A  possible 
explanation  is  that  higher  quality  standards  will  often 
result  in  a  temporary  period  of  operational  problems 
when  the  Major  Project  is  being  completed  —  i.e., 
schedule  delays  and  cost  overruns  due  to  additional  work 
being  required  and  backlog  concerns  as  high  quality 
personnel  are  transferred  from  proposal  writing 
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activities.  In  order  to  compensate  for  the  attrition  of 
high  quality  workers  that  is  likely  to  occur  as  a  result 
of  these  problems,  the  Project  Manager  will  quickly 
expand  the  workforce  when  the  Major  Project  is  brought 
under  control.  The  net  result  is  a  'larger,  but  lower 
quality  labor  force. 

As  discussed  above  higher  capture  rate  values 
result  in  increases  in  the  size  of  the  labor  force.  Due 
to  the  relative  scarcity  of  high  quality  personnel, 
however,  this  expansion  causes  a  lower  average  quality 
workforce .  This,  as  before,  is  due  to  the  temporary 
operational  problems  incurred  in  meeting  standards  and 
the  resultant  attempts  of  the  Project  Manager  to  compen¬ 
sate  for  labor  force  attrition  and  to  maintain  capacity 
by  hiring  readily  available  low  quality  personnel. 

3.3.7  Corporate  Considerations 

Cumulative  cash  flow  were  shown  to  be  significantly 
affected  by  several  initial  conditions  including  backlog 
goals  and  the  size  of  the  labor  force.  Large  values  for 
total  labor  and  higher  backlog  goals  are  associated  with 
improved  (higher)  values  of  cash  flow.  Higher  capture 
rates  for  new  proposals  have  the  expected  positive 
effect  on  cash  flow  as  do  higher  overhead  rates,  target 
costs  and  smaller  (but  larger  numbers  of)  contract 
awards.  The  proxy  for  contractor  profitability,  net 
contract  awards.  The  proxy  for  contractor  profitabili¬ 
ty,  net  cash  flow  as  a  percent  of  billings,  was 
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generally  insensitive  to  variations  in  the  independent 

variables  under  analysis. 

3 . 4  Conclusion 

The  sensitivity  analysis  phase  of  the  research  has 
provided  additional  insights  as  to  the  basic  properties  of 
DPM  behavior .  Rather  than  constraining  the  investigation  of 
DPM  performance  determinants  to  several  (pre-specif ied) 
combinations  of  policy  instrument  settings  (scenarios) ,  the 
multiple  regression  experiment  allows  for  the  simultaneous 
evaluation  of  many  independent  variables  over  wide  ranges  of 
possible  values.  A  valuable  side-effect  of  this  approach  is 
that  the  DPM  is  implicitly  subjected  to  an  array  of  consis¬ 
tency  and  plausibility  checks  which  are  necessary  and  vital 
components  of  the  validation  process. 

Due  to  the  aggregate  nature  of  regression  results, 
complex  behavioral  relationships  endogenous  to  the  DPM  are 
often  difficult  to  diagnose.  This,  of  course,  reflects  to  a 
certain  degree  on  the  nature  of  the  economic  system  being 
analyzed  and  should  not  be  considered  a  serious  deficiency. 
In  any  case,  attempts  have  been  made  throughout  Technical 
Reports  1  and  2  to  provide  explanations  of  interactions  when 
deemed  necessary .  Taken  in  combination  the  scenario  analyses 
and  the  sensitivity  analyses  provided  a  reliable  means  to 
evaluate  the  internal  logic  and  behavior  of  the  DPM 
simulation. 
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SECTION  4:  EXTERNAL  VALIDATION  INTERVIEWS* 

4 . 1  Overview 

This  section  describes  the  results  of  the  external 
validation  effort. 

The  approach  taken  was  to  conduct  interviews  with  senior 
DOD  personnel  who  have  or  had  command  responsibility  for 
major  systems  development  projects  and  with  senior  line 
managers  at  contractor  organizations. 

At  the  DOD  level  interviews  were  completed  with  General 
Bob  Mathis  (ret) ,  Rear  Admiral  Wayne  Meyer,  General  Alton 
Slay  (ret)  and  Admiral  Bud  C.  P.  Ekes  (ret)  presently  Vice 
President  Boeing  Aerospace  Company.  Contractor  personnel 
with  whom  interviews  were  completed  include  Mr.  Robert  B. 
Norris,  Manager  Operational  Planning  Military  Engine  Divi¬ 
sion,  Aircraft  Engine  Group,  General  Electric  Company,  Mr.  J. 
R.  Utterstrom,  Vice  President  and  General  Manager,  Air 
Launched  Cruise  Missile  Program,  Boeing  Aerospace  Company  and 
Dr.  Sidney  W.  Hess,  Vice  President  and  General  Manager, 
Aerospace  Division,  ICI  Americas,  Inc. 


★ 

This  section  is  adapted  from  Technical  Report  #3. 
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The  interviews  followed  a  semi  structured  open  ended 
format.  Each  interviewee  received  an  explanation  of  the 
purpose  of  the  interview  and  was  also  informed  of  the  refer¬ 
ence  person  who  recommended  that  he  be  contacted.  The 
explanation  of  purpose  described  in  general  terms  the  nature 
of  the  research  effort  to  date,  the  purpose  of  external 
validation  and  concluded  with  an  expression  of  hope  that  the 
interviewee  would  consent  to  a  free  form  discussion  and 
sharing  of  his  experience  with  the  principal  investigator. 
In  all  cases  the  individuals  contacted  agreed  to  be  inter¬ 
viewed.  However,  because  of  scheduling  conflicts  and  person— 
si  reasons  interviews  were  only  completed  with  individuals 

named  above.  The  interviews  are  being  characterized  as  semi 

\ 

structured  and  open  ended  because  for  each  group  of  inter¬ 
viewees  they  covered  a  specific  set  of  subjects.  The  order 
of  the  issues  discussed  varied  across  interviews,  because  the 
open  ended  format  led  to  the  exploration  of  additional  topics 
which  were  of  interest  to  the  interviewee  or  to  the  principal 
investigator.  The  interviews  with  Mr.  Norris,  Mr.  Utterstrom 
and  Dr.  Hess  were  held  in  person.  The  interviews  with 
General  Mathis ,  Rear  Admiral  Meyer ,  Admiral  Ekes  and  General 
Slay  were  conducted  by  telephone.  The  interviews  ranged  from 
one  to  two  hours  and  in  a  number  of  instances  involved  follow 
up  telephone  calls.  Because  several  of  the  individuals 
requested  that  findings  should  not  be  directly  attributed  to 
them  the  summary  of  the  interviews  which  follows  does  not 
explicitly  identify  any  individual.  It  should  be  noted  that 
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ths  interview  with  Admiral  Ekes  (ret)  explored  the  issues  at 
both  stages  of  his  career  (i.e.,  project  manager  Harpoon 
Missile,  Chief  CNM  and  more  recently  as  V.P.  Boeing  Aerospace 
Company) . 

4 . 2  Summary  of  Interviews  with  Senior  POD  Personnel 

The  purpose  of  the  interviews  with  the  senior  DOD 
project  commanders  was  to  explore  with  them  their  thoughts  t 

and  experiences  with  contractor  motivation,  monitoring 
project  performance,  means  of  exerting  pressure  on  contrac¬ 
tors,  importance  of  past  performance  on  award  decision, 
incentive  contracts  and  share  ratios,  role  of  IR&D,  govern¬ 
ment  weapons  procurement  policies,  and  perceptions  of  con¬ 
tractor  internal  organization  and  management  style. 

In  general,  all  interviewees  felt  that  program  failures 
to  meet  technical  specifications,  delivery  schedules  and/or 
costs  were  rare  occurrences  and  in  the  most  spectacular  cases 
were  not  entirely  the  fault  of  the  contractor.  Problems  on 
system  development  projects  were  attributed  to  several 
causes  -  failures  of  contractor  management,  political  factors 
(e.g .,  congressional  appropriation  process) ,  project  not  of  v 

priority  to  DOD,  insufficient  latitude  in  modifying 

i 

project  specifications,  DOD  project  monitoring  failed  to  pick 
up  problems  at  early  stages  of  project,  contractor  did  not 
have  appropriate  technical  capabilities,  adversarial  rela¬ 
tions  between  the  prime  and  the  sub  contractor,  and/or 
frequent  DOD  redirection  of  project. 
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As  to  the  aspect  of  monitoring  the  progress  of  projects, 
the  interviewees  agreed  that  cost,  schedule  and  technical 
performance  were  the  primary  factors  which  needed  to  be 
tracked  in  detail.  However,  two  interviewees  considered 
technical  performance  to  be  the  primary  criteria  and  one 
considered  cost  performance  to  be  the  most  important  criteria 
to  be  monitored. 

» 

Of  particular  interest  were  the  interviewees  descrip- 
tions  of  actions  to  be  taken  when  problems  are  identified. 
There  seems  to  be  total  agreement  that  in  the  first  instant 
problems  are  to  be  resolved  at  the  level  of  the  project 
managers  (DOD  and  contractor) .  It  also  became  clear  that  the 
likelihood  of  local  resolution  of  problems  was  a  function  of 
early  identification  and  of  the  project  manager  discretion  to 
adjust  schedules,  budgets  and  technical  specifications. 

When  it  becomes  evident  that  problems  are  not  being 
corrected  the  respondents  agreed  that  the  pressure  level  had 
to  be  escalated  up  the  hierarchy.  Each  related  specific 
examples  from  their  experiences  when  problems  had  to  be 
w  brought  to  the  attention  of  higher  level  management.  In 

addition  to  formal  communication  (through  letters  and  meet- 

i 

mgs)  Two  respondents  felt  that  on  occasion,  salvaging  a 
project  requires  changes  in  the  contractor  management  of  the 
project.  Such  changes,  they  felt,  can  only  come  about 
through  intervention  with  top  management.  The  interviewees 
cited  the  withholding  of  progress  payments  and  the  withdrawal 
of  the  award  fee  (which  the  contractor  earns  up  front)  as 
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very  effective  means  for  getting  top  management  attention. 

Examples  cited  included  symbolic  withholding  of  small  amounts 
(e.g.7  $100,000)  to  a  very  significant  withholding  of 
$40,000,000  in  a  major  case.  They  also  stated  that  with¬ 
holding  of  progress  payments  tend  to  be  more  effective  in 
aerospace  related  programs  and  less  effective  in  shipbuilding. 

It  should  be  noted  that  the  interviewees  opinion  that  con- 

» 

9 

tractors  are  extremely  sensitive  to  their  cash  flow  and  the 
withholding  of  progress  payments  are  at  variance  with  the 
views  of  DOD  project  managers  interviewed  in  Phase  1  (Con¬ 
tract  #F33615-79-C-5151)  described  in  Technical  Report  #3, 
pp.  1-4.  This  may  be  accounted  for  by  the  higher  ranks  of 
the  interviewees  or  because  of  their  direct  experience  with 
the  withholding  of  progress  payments. 

The  interviewees  disagreed  on  how  to  structure  the 

*  * 

contractual  relationship.  Two  respondents  felt  strongly  that 
it  is  more  effective  to  accomplish  systems  development 
projects  on  a  cost  plus  fee  basis  and  that  it  is  an  impossi¬ 
bility  to  write  "perfect"  contracts  with  "perfect"  specifi¬ 
cations.  In  the  words  of  one  interviewee,  "complex  v 

undertakings  must  have  a  lot  of  elastic".  Whereas,  the  above 

<k 

sentiment  was  also  voiced  by  the  other  interviewees  they 
felt,  to  varying  degrees,  that  firm  fixed  price  contracts  or 
incentive  contracts  with  punitive  share  ratios  are  effective 
and  can  be  written. 

It  was  clear  that  the  respondents  voiced  basic  disagree¬ 
ments  regarding  the  use  of  different  contractual 
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relationships.  One  respondent  argued  that  competitively 
awarded  fixed  price  contracts  lead  to  contractor  ripoffs  of 
the  DOD  (in  terms  of  technical  guality  and  ease  of  service 
and  maintenance)  which  inevitably  lead  to  higher  life  cycle 
costs.  Another  respondent  felt  that  firm  fixed  price  con¬ 
tracts  "incentivise"  the  contractor  on  project  costs  and 
therefore  it  should  not  come  as  a  surprise  that  the  contrac¬ 
tor  compromises  the  technical  requirements  of  the  project. 
This  same  respondent  preferred  a  cost  plus  award  fee  type 
contract,  with  the  award  fee  tailored  to  the  achievement  of 
technical  and  cost  milestones. 

As  to  incentive  type  contracts  the  interviewees  were  in 
complete  agreement  that  unless  the  share  line  is  punitive 
(40-60%)  and  unless  ceiling  costs  are  within  25%  of  target 
costs  such  contracts  will  motivate  contractors  to  exceed 
target  costs  and  tend  to  subsidize  the  size  of  engineering 
staffs . 

The  interviewees  were  in  general  agreement  as  to  the 
role  of  IR&D  funds  in  furthering  technological  innovation. 
One  respondent  described  IR&D  funds  as  "high  leverage  dollars 
to  fuel  applications  research".  Another  made  a  causal 
connection  between  IR&D  funds  and  the  flow  of  new  ideas  from 
contractor  laboratories  to  the  DOD.  And  another  felt  that  U. 
S.  technological  superiority  in  weapon  systems  could  be 
traced  to  the  tradition  of  IR&D  funds  and  the  competition 
which  it  fostered  between  vendors  for  new  ideas  and  for  the 
application  of  technology. 
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The  interviewees  also  provided  many  miscellaneous 
comments  and  opinions.  One  interviewee  made  a  distinction 
between  shipbuilding  projects  which  are  built  one  at  a  time 
(i.e.,  custom  products)  and  aerospace  projects  which  are 
built  in  series.  He  felt  that  cost  overruns  and  failure  to 
achieve  technical  specifications  are  more  likely  in  ship¬ 
building  programs  because  of  great  sensitivity  to  engineering 
change  orders.  In  aerospace  projects  the  learning  curve 
invariably  should  lead  to  the  achievement  of  increasingly 
higher  technical  specifications  and  better  costs. 

Another  respondent  noted  that  in  aerospace  projects  the 
contractor  is  responsible  for  all  phases  of  the  project  from 
design  to  production.  Whereas  shipbuilders  are  primarily 
constructors,  building  to  specifications  prepared  by  naval 
architects  which  implement  designs  originated  by  the  Navy. 

As  a  result,  shipbuilders  are  less  involved  in  a  project. 
Success  or  failure  does  not  reflect  on  their  professional 
capabilities  in  design,  technological  innovation  of  weapon 
systems  or  their  application. 

The  interviewees  were  all  of  the  opinion  that  contractor 
past  performance  should  be  factored  into  the  award  decision 
of  a  new  contract.  They  stated  that  past  performance  had 
been  considered  on  projects  they  were  involved  with.  Howev¬ 
er,  they  also  felt  that  the  source  selection  panel  "has  to  be 
subjective"  in  its  consideration  of  past  performance  and  that 
the  source  selection  is  influenced  by  past  association  with 
contractors.  One  respondent  noted  that  the  services  have, 
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over  the  years,  developed  close  working  relationships  with 
certain  contractors  which  leads  to  "rationalizing  past 
performance"  and  "favoring"  of  incumbent  contractors  when  new 
awards  are  made. 

On  balance  the  interviews  with  senior  DOD  project 
managers  closely  reflect  the  interviews  conducted  with  DOD 
project  managers  in  Phase  1  which  were  summarized  in  Techni¬ 
cal  Report  #3.  The  more  senior  officers,  interviewed  as  part 
of  the  external  validation  effort,  were  more  explicit  about 
the  effectiveness  of  withholding  progress  payments  as  a  means 
to  getting  corporate  attention  when  serious  problems  remain 
uncorrected.  The  senior  officers  were  much  more  explicit  in 
their  views  that  defense  contractors  are  organized  to  manage 
their  cash  flow.  Similarly  the  senior  officers  supported 
earlier  statements  about  the  role  of  IR&D  funds. 

It  was  of  interest  to  note  that  the  senior  officers  were 
in  disagreement  regarding  structuring  of  the  contractual 
relationship.  The  DPM  results  for  example  suggested  the  use 
of  cost  plus  fee  type  contracts  for  research  and  development 
type  contracts  up  to  a  point  where  reliable  estimates  can  be 
made  of  the  technological  gap  which  remains  to  be  overcome. 
At  that  point  switching  to  firm  fixed  price  incentive  con¬ 
tracts  was  the  preferred  alternative.  This  concept  was 
supported  by  two  respondents. 

The  DPM  results  also  indicated  that  for  incentive  type 
contracts  to  have  an  impact  on  contractor  management  of 
projects,  the  share  ratios  must  be  more  punitive.  Three  of 
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the  senior  officers  explicitly  supported  the  idea  of  share 
ratios  in  the  40-60%  range.  One  respondent  also  felt  that 
"ceiling"  cost  should  not  exceed  125%  of  "target"  cost. 

The  interviews  also  highlighted  apparent  differences 
between  aerospace  projects  and  shipbuilding  projects  and  by 
indirection  a  distinction  between  research  and  development  > 

and  production  projects.  The  structure  of  the  DPM  best  fits 

A 

aerospace  system  development  projects.  Indeed  it  was  not 
designed  to  simulate  either  "one  at  a  time"  projects  such  as 
shipbuilding  nor  series  production  programs  such  as  missiles 
or  fighter  planes.  In  addition  the  DPM  does  not  attempt  to 
simulate  project  success  or  failure  which  are  attributed  to 
operating  style,  organization  design  and  manage¬ 
ment  quality  of  contractor  organizations.  Neither  does  the 
DPM  make  distinctions  between  prime  and  sub  contractors. 

4 • 3  Summary  of  Interviews  with  Contractor  General  Managers 
The  interviews  with  contractor  senior  managers  followed 
the  same  pattern  and  had  the  same  basic  purpose  as  those  with 
senior  DOD  personnel. 

With  the  exception  of  one  interviewee,  it  was  clear  that 

V 

cash  flow  is  ap.  important  financial  measure  which  division 
* 

managers  track  by  project.  In  addition,  these  managers  J 

were  also  tracking  new  contracts  (backlog),  indirect  costs, 
management  ratios,  and  capital  investment  projects  (including 
IR&D  projects)  . 

*  ' 

Project  Manager  in  the  DPM 
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One  manager  was  quite  explicit  in  noting  that  the 
division  was  not  practicing  shifts  in  personnel  from  direct 
to  indirect  categories  and  vice  versa  as  a  means  to  managing 
cash  flow.  The  clear  implication  was  that  the  division  was 
targeted  in  the  corporate  strategic  plan  for  investment  and 
therefore  cash  flow  or  division  profits  in  the  short  run  were 
immaterial.  It  was  implied  that  if  the  division  had  been 
designated  a  mature  business,  corporate  management  in  all 
probability,  would  have  been  concerned  with  cash  flow  and 
bottom  line  contribution. 

All  the  managers  indicated  a  preference  for  contracting 
development  projects  on  a  cost  plus  fee  basis.  One  manager 
thought  it  reasonable  to  contract  early  phases  of  development 
on  a  cost  plus  fee  basis  and  on  a  fixed  price  incentive  basis 
for  the  concluding  phases.  One  manager  was  not  willing  to 
discuss  alternative  incentive  contracts  involving  greater 
sharing  of  cost  overruns  as  a  function  of  various  sharing 
proportions  and  target  and  ceiling  costs.  The  other  three 
managers,  in  general,  agreed  that  low  cost  sharing  formulas 
(in  the  10-20%  range)  did  not  create  incentives  to  minimize 
costs  and  more  likely  subsidized  excessive  staffs  and  other 
costs  (e.g.,  data  processing).  Two  managers  noted  that  high 
sharing  proportions  and/or  low  ceilings  relative  to  target 
costs  would  lead  to  more  realistic  bidding  and  therefore  to 
lower  cost  overruns.  One  manager  thought  that  most  cost 
overruns  occur  as  a  result  of  deliberate  low  bidding  and/or 
because  of  the  inducements  built  into  the  incentive  contract 
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P^bsmeters  (low  sharing  ratios  and  high  ceilings  relative  to 
target) . 

As  to  relations  with  DOD  project  managers  the  general 
sentiment  was  that  it  was  best  to  resolve  problems  at  the 
project  level.  Two  managers  felt  that  whether  a'  problem  got 
resolved  or  became  a  major  issue  was  a  function  of  the  > 

latitude  built  into  the  contract  specifications.  One  manager 
gave  the  example  of  test  firings  where  the  contract  was 
behind  on  the  number  of  test  firings.  However,  the  DOD 
project  manager ,  accepted  the  project  because  all  the  test 
firings  which  were  made  (75%  of  total  required)  exceeded 
performance  specifications. 

Another  manager  cited  an  example  of  a  missile  engine 
project  which  was  one  year  behind  schedule.  Yet  because  of 
an  ability  to  adjust  costs  and  schedules  the  project  was  not 
considered  to  be  in  "default". 

The  managers  considered  IR&D  funds  as  a  strategic  pool 
of  resources  with  a  direct  impact  for  experimenting  with  new 
ideas,  proposing  new  or  follow-on  projects  or  for  bidding  on 
new  projects.  One  manager  characterized  IR&D  projects  as  the  ¥ 

bo  his  firm  maintaining  technological  superiority. 

Another  described  in  detail  the  corporate  process  of  commit¬ 
ting  research  and  development  funds  and  the  importance 
attached  to  the  decision  allocating  corporate  funds  to 
augment  IR&D  funds. 

The  managers  were  explicit  as  to  the  transferability  of 
military  research  and  development  to  commercial  use. 
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Specific  examples  were  given  in  terms  of  jet  engine  develop¬ 
ment,  transport  aircraft  and  explosives  technology.  One 
manager  felt  that  the  transfer  of  commercial  technology  to 
aerospace  military  related  applications  was  immaterial. 

The  managers  did  express  dissatisfaction  with  DOD 
unwillingness  to  create  appropriate  risk/reward  situations 
especially  when  companies  put  up  their  own  risk  capital  for 
plants  and  equipment.  Another  manager  made  a  comparison  to 
electric  utilities  who,  he  believed,  were  guaranteed  by 
public  utility  commissions  higher  rates  of  profitability 
under  less  risk  than  what  the  DOD  allows  its  contractors 
under  conditions  of  greater  risk. 

One  manager  also  felt  that  contractors  were  at  a  disad¬ 
vantage  when  bidding  on  a  project  where  a  "special  relation¬ 
ship  does  not  already  exist  between  the  military  service  and 
the  company.  He  expressed  the  view  that  as  long  as  special 
lstionships  dominate,  "past  performance"  will  not  become  a 
significant  factor  in  the  award  decision.  It  was  also  clear 
that  the  "feast  or  famine"  approach  to  procurement  of  defense 
systems  and  the  "politics  of  defense"  were  viewed  as  major 
factors  adding  to  the  perceived  risk  of  being  in  the  military 
systems  research  and  development  business  (these  sentiments 
of  course  predate  the  present  climate  for  defense  business) . 

The  interviewees  were  also  asked  to  comment  on  the  issue 
of  competition  for  scarce  technical  talent.  One  manager 
noted  that  in  a  climate  of  contraction  top  talent  had  been 
let  go  and  that  many  more  left  voluntarily.  Another  manager 
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noted  that  his  company  was  always  searching  for  highly 
trained  and  experienced  technical  personnel  and  that  turnover 
and  growth  stimulated  competition  for  talent.  However,  he 
noted  that  in  his  company  hiring  "into  overhead"  required  his 
authorization . 

4 . 4  Summary  and  Other  Feedback  V 

During  the  three  years  of  this  research  project  the 
investigators  have  had  occasion  to  present  various  aspects  of 
the  research  at  national  meetings  of  TIMS/ORSA,  two  Winter 
Simulation  conferences,  an  AFBRMC  Workshop  on  Contractor 
Motivation,  and  to  the  DOD  Comptroller,  Dr.  Jack  Borsting  and 
members  of  his  staff.  In  addition,  the  principal  investiga¬ 
tor  and  Major  Robert  F.  Golden  (AFBRMC/RDCB)  on  July  6,  1981 
briefed  General  Joseph  H.  Connolly  (HQUSAF/RDC)  at  the 
Pentagon  on  the  research  and  its  potential  for  evaluating 
alternative  procurement  policies. 

The  value  of  presentations  at  professional  meetings  is 
the  informal  give  and  take  and  the  focus  on  potential 
applications,  limitations  of  the  model,  suggestions  for 
further  research  and  comparisons  to  prior  research.  The  t 

interest  of  Dr.  Borsting  and  his  staff  and  General  Connelly 
seemed  to  center  on  the  potential  application  of  the  DPM  in 
policy  making  involving  system  development  and  acquisition. 

Of  special  interest,  to  them,  were  the  findings  that 
suggest  the  need  for  structuring  incentive  contracts  embody¬ 
ing  significant  cost  overrun  sharing  proportions  (40%  -  70%) 
combined  with  relatively  low  ceilings  (110%  -  125%  of  target 
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costs) .  These  findings  are  consistent  with  the  validation 
interviews.  After  the  fact  it  seemed  intuitively  reasonable 
to  link  cost  overruns  to  the  structure  of  cost  incentive 
con^^’s.cts  which  in  reality  place  a  low  economic  penalty  on 
underestimating  total  costs  and/ or  which  provide  the  incen¬ 
tives  to  build  technical  capabilities  at  a  significant 
subsidy. 

Similarly  the  findings  which  support  a  stable  moderately 
increasing  weapons  development  and  acquisition  program  in 
contrast  to  an  uncertain  and  fluctuating  demand  by  DOD  also 
seemed  to  be  intuitively  appealing.  Thus  a  certain  credibil¬ 
ity  can  be  attributed  to  the  overall  model  and  to  its  poten¬ 
tial  use  in  assisting  policy  makers.  However,  the  DPM  model 
as  presently  structured  is  more  appropriate  for  system 
development  projects  such  as  new  missiles,  new  jet  engines, 
new  radar  and  new  fighter  planes,  than  for  "one  at  a  time" 
projects  such  as  shipbuilding  or  series  production.  This  is 
because  the  structure  of  the  model  includes  specific  sub 
models  for  proposing  and  securing  new  R&D  business  and 

projects.  The  DPM  model  does  not  include  a  manufac¬ 
turing  assembly  sub  model. 

Researchers  and  other  scholars  have  noted  that  the  DPM 
needs  to  be  further  validated.  While  the  overall  approach  to 
the  design  of  the  DPM  seems  supportable,  questions  have  been 
raised  as  to  the  validity  of  certain  behavioral  attributes 
which  have  been  incorporated  into  the  model  and  the  absence 
of  others  (e.g.,  adaptive  learning  by  key  decision  makers) . 
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On  balance ,  however  ,  the  model  and  its  findings  appear 
reasonable  and  lends  further  support  to  a  behavioral  alterna¬ 
tive  of  the  traditional  economic  theory  of  the  firm. 

Any  external  validation  of  the  type  attempted  in  this 
case  has  certain  limitations.  The  managers  being  interviewed 
represent  a  convenience  sample.  It  is  also  difficult  to 
arrange  meetings  with  the  appropriate  policy  makers.  In 
addition,  there  is  always  a  certain  reluctance  on  the  part  of 
managers  to  discuss  certain  issues.  Similarly  they  are 
motivated  to  describe  their  firm  or  their  project  management 
philosophy  or  practices  in  favorable  terms.  Nevertheless, 
many  of  the  comments  are  clearly  supportive  of  basic  features 
incorporated  in  the  DPM.  Most  notably  the  sensitivity  to 
pressures  (from  DOD  project  managers  or  corporate  manage¬ 
ment)  ,  importance  of  withholding  progress  payments,  backlog 
and  spinoff  projects,  competition  for  scarce  technical 
personnel  and  the  influence  of  traditional  incentive  contract 
on  cost  efficiency.  The  interviews  also  confirmed  that 
incentive  contracts  embodying  high  cost  overrun  sharing 
proportions  with  low  ceilings  relative  to  target  costs  would 
lead  to  more  realistic  bidding  and  therefore  to  fewer  cost 
overruns .  However,  it  is  not  clear  from  the  interviews 
whether  such  contracts  would  result  in  overall  higher  cost 
efficiencies  and  therefore  lower  total  costs.  The  interviews 
also  supported  the  preference  (and  effectiveness)  of  cost 
plus  fee  development  contracts. 
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The  corporate  interviewees  and  one  senior  DOD  officer 
fslt  that  fluctuations  in  DOD  demand  is  a  major  cause 
for  poor  project  performance,  underestimation  of  costs  by 
bidders  and  overselling  of  achievable  technical  specifica¬ 
tions.  The  results  of  the  DPM  simulation  clearly  lend 
i  support  for  a  stable  moderately  increasing  acquisition 

33  3  means  to  achieving  improved  project  performance, 

* 

reduce  contractor  concern  with  survival  and  increase  cost 
efficiencies . 

The  interviews  yielded  conflicting  information  on  the 
issue  of  management  concern  with  cash  flow.  Maximizing  cash 
flow  is  an  important  behavioral  characteristic  of  the  DPM. 
There  was  an  evident  reluctance  to  discuss  internal  practic¬ 
es,  yet  the  strong  impression  was  created  by  the  respondents 
that  cash  flow  considerations  were  important.  If  additional 
validation  efforts  are  to  be  undertaken  it  should  be  feasible 
to  design  a  survey  questionnaire  aimed  at  a  larger  sample  of 
contractor  and  DOD  managers.  Such  a  survey  would  draw  on  the 
insights  gained  from  the  interviews  and  should  also  probe  the 
various  policy  implications  derived  from  exercising  the  DPM. 

X 
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SECTION  5:  CONCLUSION 

The  major  practical  use  of  building  a  DPM  type  simula¬ 
tion  is  its  ultimate  application  in  answering  "what  if"  type 
policy  questions  involving  the  design  parameters  of  the 
^^^hrac tual  relationship  between  the  DOD  and  defense  contrac¬ 
tors.  For  example,  the  simulation  results  indicated  that  i 

increasing  the  contractor's  fee  improves  cost  control  perfor- 
mance.  This  idea  has  recently  been  implemented  by  the  Navy 
with  measurable  success.  Another  example  involves  the  weight 
given  to  past  performance  in  awarding  new  contracts.  The  DPM 
simulation  results  suggest  that  the  higher  the  weight  (in¬ 
cluding  those  assigned  to  the  quality  of  the  proposal)  the 
better  is  cost  control  performance  and  social  efficiency. 

Factoring  past  performance  into  the  contract  award  decision 
was  supported  by  senior  DOD  officers.  The  practical  implica¬ 
tions  involve  the  feasibility  of  factoring  past  performance 
measures  into  a  DOD  contracting  environment  where  often  it  is 
difficult  to  maintain  a  second  source  and  where  such  evalua¬ 
tion  is  considered  to  be  extremely  subjective. 

The  research  effort,  in  our  view,  clearly  demonstrates 
the  value  of  such  a  simulation  model  for  analyzing  alterna¬ 
tive  acquisition  policies  (e.g.,  the  importance  of  maintain¬ 
ing  stable  demand,  weighting  past  performance,  etc.)  and  the 
design  of  contract  incentives  (size  of  fee,  schedule  of 
progress  payments,  target  and  ceiling  costs  and  sharing 
proportions) .  In  particular  it  could  augment  traditional 
analytical  and  intuitive  approaches  to  these  issues.  Its 
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routine  use  in  this  mode,  by  the  Air  Force,  will  depend  on 
implementing  its  use  within  the  relevant  command  responsible 
for  acquisition  policies  and  the  design  of  incentive 
contracts . 

A  simulation  model  of  this  type  has  other  uses.  Of 
particular  interest  should  be  its  potential  use  in  the 
training  or  education  of  policy  makers  and  or  DOD  project 
managers.  In  business  education  similar  simulation  models 
have  been  designed  as  management  games.  Such  games,  which 
can  be  extremely  complex,  are  used  as  laboratories  for 
training  students  to  apply  analytical  tools  and  integrate 
functional  areas  knowledge  (marketing,  production,  account¬ 
ing,  financial  planning,  etc.)  within  a  competitive  decision 
making  environment.  The  DPM  simulation  has  the  potential  to 
be  integrated  into  an  "Acquisition  Game"  and/or  a  "Project 
Management  Game"  for  project  managers.  It  could  also  be  used 
in  the  education  of  policy  makers  (e.g.,  Congressmen,  senior 
staffs)  when  they  need  to  develop  insights  into  DOD  acquisi¬ 
tion  policies.  The  current  state  of  the  DPM  simulation 
cannot  be  immediately  used  in  acquisition  training.  However, 
it  should  be  at  the  core  of  an  acquisition  game  which  could 
be  designed. 

In  summary,  the  total  research  effort  has  produced  a 
simulation  model  involving  the  DOD  and  the  contractor  organ¬ 
ization  at  the  corporate  and  project  management  levels.  The 
simulation  has  been  shown  to  be  internally  robust  and  to 
satisfy  certain  face  validity  criteria.  In  particular  its 
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potential  application  to  acquisition  related  policy  making 
has  been  demonstrated.  It  is  in  the  area  of  contributing  to 
policy  making  and  project  management  training  where  future 
developments  need  to  be  directed. 
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